Video and still image data alteration to enhance privacy

ABSTRACT

A computer alters at least one recognizable metric or text in a digitally encoded photographic image by operating an alteration algorithm in response to user input data while preserving an overall aesthetic quality of the image and obscuring an identity of at least one individual or geographic location appearing in the image. An altered digitally-encoded photographic image prepared by the altering of the at least one recognizable metric or text in the image is stored in a computer memory. User feedback and/or automatic analysis may be performed to define parameter values of the alteration algorithm such that the alteration process achieves preservation of aesthetic qualities while obscuring an identity of interest.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and is a continuation of U.S. patentapplication Ser. No. 16/292,216, filed Mar. 4, 2019 (now U.S. Pat. No.11,080,513, to be issued Aug. 3, 2021), which is a continuation-in-partof U.S. patent application Ser. No. 15/665,216, filed Jul. 31, 2017(U.S. Pat. No. 10,223,576, issued Mar. 5, 2019), which is a continuationof U.S. patent application Ser. No. 14/853,782, filed Sep. 14, 2015(U.S. Pat. No. 9,721,144, issued Aug. 1, 2017), which is a continuationof U.S. patent application Ser. No. 14/294,047, filed Jun. 2, 2014 (U.S.Pat. No. 9,135,493, issued Sep. 15, 2015), which is a continuation ofU.S. patent application Ser. No. 13/349,546, filed Jan. 12, 2012 (U.S.Pat. No. 8,744,119, issued Jun. 3, 2014), which claims priority to U.S.Provisional Patent Application Ser. No. 61/431,965, filed Jan. 12, 2011,all of which are hereby incorporated by reference in their entireties.

FIELD OF INVENTION

This application relates to methods and systems for image dataprocessing and image sharing over a network.

DESCRIPTION OF THE BACKGROUND

Anonymity on the internet is being destroyed by technology that allowssearching databases of images for people. There are services such as“tineye” that find duplicate photographs, and technology such asGoogle's Picasa are capable of going much further, identifying a personin a photograph and then finding them in other photographs. For example,a Congressman's swearing-in photo might be used to develop the set offacial characteristics that are then looked for in other pictures,leading to the location of otherwise obscurely located embarrassingbachelor party pictures. See also the United States patent on thesubject, “Method and Apparatus for Photograph Finding”, U.S. Pat. No.7,454,498 and related continuations. It would be desirable for peoplewho publish personal images on computer networks to conveniently ensurethat posted images will not be subject to undesired image recognitionapplications, but solutions to provide such protection while stillenabling online sharing of personal image data are either currentlyunavailable or suffer from substantial limitations in convenience oreffectiveness.

In addition to recognizing elements of still images, automated analysisof moving images can be used to identify a person via techniques such asgait recognition. One advantage of using body shape, characteristicsand/or gait as a basis for identification of a person in a moving imageis that the resolution necessary for recognition is reduced, allowingidentification of persons at a far greater distance, with inferiorfocus, or in the background of a video.

SUMMARY OF THE INVENTION

Methods and systems for image processing to protect specified personalidentities or geographic locations from automatic recognition, orgraphic data alteration to enhance online privacy, are described indetailed in the detailed description, and certain aspects are describedor summarized below. This summary and the following detailed descriptionshould be interpreted as complementary parts of an integrateddisclosure, which parts may include redundant subject matter and/orsupplemental subject matter. Omissions in either section do not indicatepriority or relative importance of any element described in theintegrated application. Differences between the sections may includesupplemental disclosures of alternative embodiments or additionaldetails, or alternative descriptions of identical embodiments usingdifferent terminology, depending on the nature of the differences.

In an aspect, a method for graphic data alteration to enhance onlineprivacy may include a number of steps, all of which may be performed bya computer, either automatically or semi-automatically. The method mayinclude altering at least one recognizable metric or text in adigitally-encoded photographic image by a computer operating analteration algorithm in response to user input data while preserving anoverall aesthetic quality of the image and obscuring an identity of atleast one individual or geographic location appearing in the image. Themethod may further include storing, in a computer memory, an altereddigitally-encoded photographic image prepared by the altering of the atleast one recognizable metric or text in the digitally-encodedphotographic image. Altering the image may include altering selectedfacial recognition metrics. In addition, or in the alternative, alteringthe image may include scrambling text.

As used herein, an “aesthetic quality” means a quality determined by thesubjective taste of an individual human being, or collectively by anaggregate of subjective tastes of a group of individual human beings. Anaesthetic quality may be measured using an input-output processingmachine, such as a computer, collecting user input from user inputdevices that is correlated to machine outputs such as graphic displaysof a set of images under consideration. A heuristic aesthetic qualitymeasuring algorithm may be trained by human inputs to recognize valuesfor a set of image parameters that an individual person, or a group ofpersons will subjectively perceive as preserving an overall aestheticquality of an image. Such a heuristic aesthetic quality measuringalgorithm, after an initial training period, may be used to provideinput for automatic control of alterations to images so as to preservean overall aesthetic quality of the images, with or without further userfeedback, for an individual user or a group of users.

In another aspect, the method may include controlling at least oneparameter of the alteration algorithm so as to obscure the identity fromautomatic recognition by an image recognition algorithm to below adefined minimum confidence level. In addition, the method may includemodifying at least one parameter of the alteration algorithm in responseto user input specifying the desired confidence level.

User feedback may be used to train a heuristic algorithm so as toproduce altered images that preserve a desired aesthetic quality of animage while obscuring an identity, for example a personal identity thatis recognizable using a facial recognition algorithm. Accordingly, amethod may include publishing the altered image in an electronic medium,and receiving user feedback data, by a computer server via acommunications network, regarding at least one of whether (a) theoverall aesthetic quality of the image is preserved by the altered imageor (b) the identity is obscured in the altered image. The method mayfurther include modifying at least one parameter of the alterationalgorithm in response to the user feedback data. The method may furtherinclude repeating the altering at least one recognizable metric or textin the digitally-encoded photographic image, based on a modifiedparameter of the alteration algorithm from the modifying at least oneparameter of the alteration algorithm.

In another aspect, the method may include defining an alterationalgorithm for the altering so as to obscure the identity from automaticrecognition using a minimal value of

$\Delta = {\sum\limits_{1toM}\overset{\_}{\partial i}}$

wherein ∂l is a measure of metric alteration each of for a number M ofdifferent facial recognition metrics. The method may further includeselecting the number M of metrics so as to lower each applied ∂l tobelow a human-perceptible threshold. For example, the number M may beincreased, thereby causing an increased number of alterations on aimage, while the amount of each alteration is decreased. To use a simpleexample, instead of changing an amount of separation between eyes on aface by at least 10% to prevent automatic facial recognition (i.e., M=1and ∂l>±10%), the alteration may change an amount of separation betweenthe eyes by a lesser amount, and may also change some other metric, forexample the width of the mouth (e.g., M=2 and ∂l<±10%).

In an alternative aspect, the method may include identifying arecognition parameter set based on the recognition algorithm, whereinthe recognition parameter set is defined by a set of parameter rangevalues outside of which the recognition algorithm cannot recognize theidentity above the defined minimum confidence level. In such case, themethod may further include defining parameter values for the alterationalgorithm used in the altering of the image within the recognitionparameter set. In addition, the method may further include controllingat least one parameter of the alteration algorithm so as to preserve anoverall aesthetic quality of the image as determined by an aestheticquality measuring algorithm to above a defined minimum preservationlevel. The aesthetic quality measuring algorithm may be a heuristicallytrained algorithm that has previously been trained by user feedback overa sample set of images. The method may further include identifying anaesthetic preservation parameter set based on the aesthetic qualitymeasuring algorithm, wherein the aesthetic preservation parameter set isdefined by a set of parameter range values outside of which theaesthetic quality measuring algorithm determines that the aestheticquality of the image has changed in an amount that exceeds the definedminimum preservation level. The method may further include determining athird set of parameter values by a set difference of the aestheticpreservation parameter set and the recognition parameter set, whereinthe set difference is defined as parameter values that are members ofthe aesthetic preservation parameter set and are not members of therecognition parameter set.

In related aspects, an apparatus for graphic data alteration to enhanceonline privacy may include a processor coupled to a memory and a networkinterface, the memory holding instructions that when executed by theprocessor cause the apparatus to perform any of the methods and aspectsof the methods summarized above or described elsewhere herein. Certainaspects of such apparatus (e.g., hardware aspects) may be exemplified byequipment such as a network interface for network communications.Similarly, an article of manufacture may be provided, including anon-transitory computer-readable storage medium holding encodedinstructions, which when executed by a processor, may cause a specialtydevice configured as an image processing server or client node toperform ones of the methods and aspects of the methods described herein.

Further embodiments, aspects and details of the method and apparatus forgraphic data alteration to enhance online privacy are presented in thedetailed description that follows.

BRIEF DESCRIPTION OF THE DRAWINGS

The present technology, in accordance with one or more variousembodiments, is described in detail with reference to the followingfigures. The drawings are provided for purposes of illustration only andmerely depict typical or example embodiments of the technology. Thesedrawings are provided to facilitate the reader's understanding of thetechnology and shall not be considered limiting of the breadth, scope,or applicability of the technology.

FIG. 1 is a block diagram showing an example of a system for graphicdata alteration to enhance online privacy and related operations.

FIG. 2 is a diagram illustrating concepts related to facial recognition.

FIG. 3 is a flow chart showing an example of a method for graphic dataalteration to enhance online privacy.

FIG. 4 is a diagram illustrating aspects of set operations fordetermining parameters of an image alteration algorithm for preservingan overall aesthetic quality of an image and obscuring an identity of atleast one object appearing in the image.

FIG. 5 is a block diagram illustrating aspects of an apparatus forperforming a method as shown in FIG. 3.

DETAILED DESCRIPTION

To improve anonymity, an apparatus may process image data to defeat theability of automated programs to identify faces or other elements. Imagedata may be provided in a great variety of formats, for example, TIFF,GIF, JPEG, MPEG, and so forth, which may include both still images orvideo frames. The present technology is not limited to a particularimage data format. The present technology may include a method whereinthe same criteria utilized by a computer performing automatic analysisof images to identify faces (i.e. distance between pupils, distance fromtip of nose to ears, similar metrics) to trick the identificationsoftware. The technology may include a image alteration method similarto those used in Photoshop's™ “Puppet Warp”, “Liquify”, or othermanipulation techniques to introduce minor modifications to the facialor other photographic characteristics, for example body characteristicsor even vehicle, house, or other object characteristics when attemptingto keep such photographs “private”. In such methods, the amount ofalteration may be automatically controlled according to techniques asdisclosed in more detail herein to achieve anonymity without sacrificinga desired overall aesthetic quality for an image. In an example, onemight take a photograph of a 9 year old daughter, submit it to theprogram, elect a “moderately aggressive” obscuration level, and theprogram would slightly enlarge the distance between her eyes, reduceslightly her nose and ear size, slightly shorten her chin and plump herlips. The more aggressive the modifications, the more likely the changeswould cause a noticeable (to humans) alteration in appearance, but thebillions of photographs online provide such a large database to compareto that even changes small enough to be imperceptible to the human eyemay often be sufficiently significant to a computer program to cause itto be unable to identify an altered image as being of the same person asan unaltered image. Alterations may need to be more aggressive ascomputing power and recognition algorithms improve, or if there is aneed to defeat recognition within a smaller subset of photos, forexample, within a geocoded group of photos taken within a limitedgeographic area. The aggressiveness of the alterations may be setmanually, or automatically depending on the metadata or other indicia ofuniqueness within the photograph.

To add to the anonymity, text accompanying photographs can optionally bescrambled and made non-searchable by scrambling the middle letters.Humans are capable of reading sentences where the letters in each wordare all present, the first and last letter are in proper position, butthe remaining letters are randomized. For example, “Hnumas are calabpleof rdeanig seetnnces whree the leettrs in ecah wrod are all psenret”.This may be done in conjunction with facial alteration, or independentlyof it.

In addition, identical alteration schemes may be applied for each of agroup of photos. This way, for example, all public photos of CongressmanSmith might be identically altered (i.e. eye distance increased by 4%,nose size reduced by 2%, etc), so that photos within a group would beassociated by automated identification software. One searching forCongressman Smith photos would then be able to find all of the alteredphotos as identified with appropriate text or tags in the contextual webpage, but would not be able to find the unaltered “frank smith” photos.This kind of grouping would reduce the chance that an unaltered photowould be seen as similar to an altered photo. That is, a set of randomlyaltered photos might fit into a resulting continuum of alterations donearound a common feature set, namely the metrics of the originalunaltered photo, thereby increasing the risk that that the photos couldat least in theory be identifiable as belonging to the same group. Byapplying systematic alterations to a group of photos, a photo publishermay in essence generate a new “median” common feature set for the personat issue. As used herein, a “scheme” means an algorithm including a setof parameters that may be varied to control how the algorithm operates,and may sometimes be used interchangeably with “algorithm.”

FIG. 1 shows a system 100 including a client or user interface component106 connected to an image processor 102, directly or via a WAN cloud110. The processor 102 may be, or may include, one or more computerprocessors, for example multi-core or single-core processors sold undervarious product names from suppliers such as Intel™, AMD™ and others.The user interface component may receive digital signals from a userinterface device 108, for example a touchscreen, keyboard, camera ormicrophone, derived from physical user input. The image processor 102may comprise various components or modules 114, 116 and 118, which maybe coded in hardware, firmware, software, or some combination of theforegoing. Accordingly, the modules 114, 116 and 118 may be, or mayinclude, a computer memory or non-transitory computer-readable mediumholding instructions executable by the processor 102 to cause an imagealteration apparatus to perform one or more operations of the methodsdisclosed herein. The modules 114, 116 and 118 coupled to the processor102 may include means for image analysis, means for defining analteration scheme, and means for image alteration. Each of said meansmay include the processor 102 operating an algorithm, according to moredetailed aspects of algorithms and/or data processing or input/outputoperations as disclosed elsewhere herein. In addition, the processor 102may receive information from a de-constructor module 124, which may beincluded in programming for the processor or linked to it via a linkfrom an external device.

One or more digitally encoded photographic images 104 from any suitablesource may provide input for system 100. An image 104 may be provided tothe processor 102 by any suitable method, for example by uploading froma remote client or downloading from a remote storage or cloud storageresource. Generally, the image should be unpublished to a public forumat the time it is provided to the processor 102 for alteration, becausea purpose of the alteration is to protect the privacy of persons orpersonally-identifiable objects appearing in the image 104. In anembodiment, a purpose is to prevent an automatic facial or personalrecognition algorithm from recognizing the identity of specific personsor places appearing in the image 104.

To this end, the de-constructor (i.e., reverse engineering) component124 may operate independently to determine one or more recognitionboundaries of an applicable recognition algorithm 122. “Recognitionboundaries” may be, or may include, a parameter set for applicablemetrics that is defined by a set of parameter range values outside ofwhich the recognition algorithm cannot recognize the identity above thedefined minimum confidence level. The recognition algorithm 122 maygenerally be considered to be a “black box” under the control of anentity independent of any operator of processor 102. Thus, use of thede-constructor assumes that it is not possible to directly determine howthe recognition algorithm 122 operates; for example, if source code forthe algorithm 122 is available to the operator of the image processor102, there may be no need to operate a de-construction component 124.

The deconstruction component 124 may determine recognition boundaries bysubmitting a collection of images to the algorithm, including images ofthe same person or place in which details have been altered by varyingamounts, then using the algorithm to identify matches against anunaltered photograph of a person or place. Results of the searchreturned by the recognition algorithm 122 may be analyzed to determinehow much alteration is needed to renter a person or face unrecognizableto the algorithm, optionally as a function of the amount of image datasearched. Analysis results may be compiled to prepare data for use by amodule 116 for defining an alteration scheme. De-construction may beperformed automatically, semi-automatically, or manually.

The processor component 102 may include a component 114 for performingimage analysis. The component 114 may automatically orsemi-automatically identify objects appearing in the digital image. Forexample, the photograph 102 may be digitally analyzed to identify one ormore faces appearing in the image, and then via the user interface 106solicit and receive user input selecting one or more of the faces foralteration. Similarly, the application may identify recognizable textappearing in the image 104, and receive user input via the userinterface 106 designating one or more text areas for alteration.

The processor component 102 may also comprise a component 116 fordefining an alteration scheme for altering selected portions of an image104. The alteration scheme may be defined in response to user input andusing information available for the applicable recognition algorithm ordata set size to provide a specified level of confidence that, afteralteration, the object will not be recognized by the recognitionalgorithm operating on the assumed data set and an overall aestheticquality of object will be preserved. Default present values may beprovided upon program initialization, and users may be enabled to changedefault values via the user interface 106. For example, a defaultconfidence level for non-recognition may be set at 98%, and a user maychange this to any desired value, for example 99.99% or 90%. In general,lowering the confidence level will result in less alteration of theimage and raising the confidence level will result in greater alterationof the image. Accordingly, by raising or lowering a specified confidencevalue via a user interface to the alteration scheme component 116, auser may directly control an amount of overall aesthetic quality topreserve. In addition, alteration schemes may be configured to alterimage metrics that are likely to be effective in preventing automaticrecognition algorithms from recognizing a person or place appearing in aphotograph, without causing a human who knows the person or place to beconfused—or even to notice the alteration at all. The alteration schememay include randomly selected elements or parameters, for exampleparameters randomly selected from a defined range of possibilities.

The image processor 102 may further comprise an image alterationcomponent 118 for altering the image in according with a selectedalteration scheme. Digitally this may be accomplished by translatingand/or stretching selected parts of the image to be altered using anautomatic alteration algorithm, in amounts defined by the alterationscheme, to provide an altered image 120. The altered image 120 shouldnot contain information that would enable restoration of the originalimage 104 by anyone not possessing the original. For example, thealtered image should be scrubbed of any metadata indicating whatalterations were made and should not include any tell-tale artifactsindicative of particular alterations. Such artifacts may be avoiding byusing a well-designed alteration algorithm that introduces random orquasi-random variability in altered pixel values, so that altered pixelvalues are not rigidly determined by adjoining unaltered pixelsappearing in the image 120. The altered image may be stored in a memorydevice or non-transitory computer-readable medium, and/or provided to auser-specified client 106 or server 112 via a WAN 110.

FIG. 2 provides a simple example of facial feature alteration using dualmetrics ‘X’ and ‘Y’ appearing in an unaltered facial image 200 and,after alteration, in an altered facial image 202, for a metric number MIn should be appreciated that in an actual facial alteration, dozens orhundreds of metrics may be involved, for example, M may be in the rangeof about 10 to 1000. The example metric X defines a center-to-centerdistance between the eyes and the example metric Y defining a distancebetween the upper edge of the lips and the center of the eyes. In thealtered facial image 202, pixels have been manipulated to change thedistance between the eyes X by an amount ‘∂x’ and the distance Y by anamount ‘∂y.’ The altered face may be characterized by a sum of absolutevalues of ‘i’ metric changes or M number of metrics as follows:

$\Delta = {\sum\limits_{1toM}\overset{\_}{\partial i}}$

Generally, it may be desirable to minimize the value of Δ by finding thelowest value that will effectively prevent automatic object recognitionwith the desired confidence level over the anticipated image data set,while applying alterations to a large enough M so that each alterationis very slight. That is, M may be maximized to the effective limitdefined by the applicable recognition algorithm, which includes allmetrics used for recognition, while Δ is minimized by altering eachmetric as little as possible. Of course, increasing M reducescomputational efficiency, so the alteration scheme should not increase Mbeyond that needed to prevent automatic recognition while preserving theaesthetic qualities of the image to its intended audience. In addition,color or texture of skin tone and eyes may also be altered, or text maybe scrambled as described above.

Aesthetic qualities, and a permissible amount of alteration thatpreserves overall aesthetic quality, are subjectively determined by anindividual or group. These parameters can be systematically measuredusing a heuristic learning process based on user feedback, and mayremain fairly stable over time for a given person or group. Thus, animage processing apparatus may determine a level of Δ, M and/or ∂l so asto preserve overall aesthetic quality of an image according topreferences of a person or group.

FIG. 3 is a flow chart showing an example of a method 300 for alteringan image to prevent automatic recognition. At 302, the method mayinclude receiving a digitally encoded image for alteration at aprocessor, or accessing the image identified by a user in a database. At304, the processor may analyze the image to identify objects, such asfaces, text, or locations, to be altered. Any suitable analysis methodmay be used. At 306, the processor may define an alteration scheme, toeffectively prevent automatic recognition of the object to be alteredwithout noticeably impairing the aesthetic of the image for its intendedaudience. For example, the processor may operate to apply a minimumeffective Δ over a suitable large number M of metrics. In addition, orin the alternative, the processor may determine parameter sets forrecognition boundaries and quality preserving boundaries, and base thealteration scheme on a set difference of the aesthetic qualitypreserving parameters and the recognition boundary parameter values.

Independently, the processor or a different processor may identify 312one or more automatic facial recognition algorithms that are to bedefeated, and using an iterative testing process, identify recognitionboundaries 314 of the applicable algorithm. These boundaries may beparameterized and used for defining the alteration scheme 306. Theoperation summarized at block 314 may include identifying a recognitionparameter set based on an object (e.g., facial) recognition algorithm,wherein the recognition parameter set is defined by a set of parameterrange values outside of which the recognition algorithm cannot recognizethe identity above a defined minimum confidence level. In such case, themethod 300 may further include at 306 defining parameter values for thealteration algorithm used in the altering of the image within therecognition parameter set.

In addition, the method 300 may further include at 306 controlling atleast one parameter of the alteration algorithm so as to preserve anoverall aesthetic quality of the image as determined by an aestheticquality measuring algorithm to above a defined minimum preservationlevel. The aesthetic quality measuring algorithm may be a heuristicallytrained algorithm that has previously been trained by user feedback overa sample set of images. The alteration scheme may adopt parameter valuesdesigned to cause the alteration to preserve a designated amount ofoverall aesthetic quality of an image while obscuring an identity of atleast one object appearing in the image to a designated recognitionprocess. The designated amount of quality preservation may be determineddirectly in response to user input, for example by receiving user inputindicating a minimum confidence level that the alteration willeffectively defeat an automatic recognition process. In the alternative,the designated amount of aesthetic quality preservation may beautomatically determined using a predictive algorithm as describedherein, optionally a heuristic algorithm, and optionally using setoperations with a recognition parameter set to define parameter valuesfor the image alteration.

In an aspect, the method 300 may further include, at 316, identifying anaesthetic preservation parameter set based on the aesthetic qualitymeasuring algorithm, wherein the aesthetic preservation parameter set isdefined by a set of parameter range values outside of which theaesthetic quality measuring algorithm determines that the aestheticquality of the image has changed in an amount that exceeds the definedminimum preservation level. The method may further include, at 306,determining a third set of parameter values by a set difference of theaesthetic preservation parameter set and the recognition parameter set,wherein the set difference is defined as parameter values that aremembers of the aesthetic preservation parameter set and are not membersof the recognition parameter set.

An example of such set relationships is presented in FIG. 4,illustrating aspects of set operations 400 for determining parameters ofan image alteration algorithm for preserving an overall aestheticquality of an image and obscuring an identity of at least one objectappearing in the image. It should be appreciated that the illustratedset operations are simplified for clarity of example, and that in thepractice of the method, the set operations would involve determiningmany more parameter values and processing many more set operations thanwould be practicable except by a computer that is programmed accordingto the teaching herein. Accordingly, it should be apparent that such acomputer should be used to perform the computational operationsdescribed herein for determining parameters of an image alterationalgorithm that preserve an overall aesthetic quality of an image whileobscuring an identity of at least one object (e.g., person or geographiclocation) appearing in the image.

A first parameter set 402 a represents a recognition boundary for aparticular recognition process, e.g., a facial recognition algorithm.The parameter set includes a set of parameter value ranges 406, 412, 418and 424 arranged around a baseline 404 a. These ranges representpictorially an amount by which parameters of an image feature (e.g., animage of a person's face) can be altered without rendering the imagefeature unrecognizable by the applicable recognition process. Each rangeof parameter values 406, 412, 418 and 424 may include values greaterthan (+) or less than (−) the baseline 404 a, which represent themeasured actual value. The baseline value 404 a is generally differentfor each parameter, and the center value and width of the range relativeto the baseline may vary for each parameter. For example, parameter 406may represent the distance between the top of the head and the lowerjaw, with a baseline value of ten inches for a particular individual;parameter 412 may represent a distance between the eyes with a baselinevale of four inches for the individual, parameter 418 may represent thedistance between the upper and lower eyelid, with a baseline value of0.5 inch for the individual, and 424 may represent ratio of face widthto length, with a baseline value of 0.8, as so forth. These examples aremerely for illustration and are not intended to represent preferred ortypical values.

A second parameter set 402 b represents an aesthetic qualitypreservation boundary set corresponding to the same parametersillustrated for set 402 a. The parameter values 408, 414, 420 and 426represent a range of values respectively corresponding to the samemeasures 406, 412, 418 and 424. However, the parameter values in set 402b are generated as aesthetic preservation parameter set based on anaesthetic quality measuring algorithm, wherein the aestheticpreservation parameter set is defined by the set of parameter rangevalues 408, 414, 420 and 426 outside of which the aesthetic qualitymeasuring algorithm or user input determines that the aesthetic qualityof the image has changed in an amount that exceeds a defined minimumpreservation level.

A third parameter set 402 c represents a set difference of set 402 b and402 a. The set difference may be defined as parameter values 410 a, 410b, 416 and 422 that are members of the aesthetic preservation parameterset 402 b and are not members of the recognition parameter set 402 a. Insome cases a parameter value difference may result in a null result, forexample where a preservation parameter value range 426 in set 402 b liesentirely within its corresponding recognition parameter value range 424in set 402 a. In such cases, an alteration scheme definition process 306may select one of the values 426 a or 426 b just outside the recognitionparameter range 424, or may select the baseline value at 404 a, or mayselect some value within the range 426. It is anticipated that using amuch larger number of parameters should usually obviate the need toalter the image with respect to every parameter, and many parameters maybe left unaltered or minimally altered to the extent not needed toobtain a desired confidence level of defeating a known maximallyeffective recognition algorithm. In selecting parameter values for imagealteration, the definition process 306 may use a Monte Carlo method orsimilar statistical/iterative process to identify a set of alterationparameter values that are most likely to defeat known recognitionalgorithms in the anticipated publication space for the altered image,while resulting in a minimal change in the image aestheticquality—thereby minimizing the perceived change in quality experiencedby the intended target audience.

Referring again to FIG. 3, at 308, the processor may alter the imageaccording to the defined alteration scheme to produce an altered imageusing an alteration algorithm, so that all metrics M selected foralteration are irrecoverably changed in the altered image. At 310, theprocessor may transmit or otherwise provide the altered image to aspecified destination, including but not limited to storing the alteredimage in a computer memory. As noted image data may be stored in a greatvariety of data formats, e.g., JPEG, TIFF, GIF, etc., and any suitableformat may be used.

With reference to FIG. 5, there is provided an exemplary imageprocessing apparatus 500 that may be configured as a client or servercomputer, or as a processor or similar device for use within thecomputer. The apparatus 500 may include functional blocks that canrepresent functions implemented by a processor, software, or combinationthereof (e.g., firmware). As illustrated, in one embodiment, theapparatus 500 may comprise an electrical component or module 502 foraltering at least one recognizable metric or text in a digitally-encodedphotographic image by an alteration algorithm while preserving anoverall aesthetic quality of the image and obscuring an identity of atleast one object appearing in the image. The component 502 may be, ormay include, a control processor coupled to a receiver and to a memory,wherein the memory holds encoded instructions for the algorithm usingparameters selected to preserve an overall aesthetic quality of theimage while obscuring an identity of at least one object appearing inthe image. The component 502 may be, or may include, a means foraltering at least one recognizable metric or text in a digitally-encodedphotographic image by an alteration algorithm while preserving anoverall aesthetic quality of the image and obscuring an identity of atleast one object appearing in the image. Said means may be, or mayinclude, the at least one control processor operating an algorithm. Thealgorithm may include receiving image data in response to a user input,receiving further user input designating one or more parameter valuesfor image alteration, retrieving parameters from a heuristic learningmodule designed to obscure the object while preserving overall aestheticimage qualities, and processing the image data to obtain altered imagedata based on selected values of the parameters. The algorithm may, inthe alternative or in addition, include one or more of the detailedoperations 312-316 discussed above.

The apparatus 500 may comprise an electrical component 504 for storingan altered digitally-encoded photographic image prepared by the alteringof the at least one recognizable metric or text in the digitally-encodedphotographic image in a computer memory. The component 504 may be, ormay include, a control processor coupled to a memory, wherein the memoryholds encoded instructions for causing the apparatus to store outputfrom the image alteration algorithm in a designated computer memory. Thecomponent 504 may be, or may include, a means for storing an altereddigitally-encoded photographic image prepared by the altering of the atleast one recognizable metric or text in the digitally-encodedphotographic image in a computer memory. Said means may be, or mayinclude, the at least one control processor operating an algorithm. Thealgorithm may include receiving image data processed by the imagealteration algorithm, configuring the altered data according to aselected file format, and storing a resulting image file in a selectedcomputer memory.

In related aspects, the apparatus 500 may optionally include a processorcomponent 510 having at least one processor. The processor 510 may be inoperative communication with the components 502-504 via a bus 512 orsimilar communication coupling. The processor 510 may effect initiationand scheduling of the processes or functions performed by electricalcomponents 502-504 and similar components.

In further related aspects, the apparatus 500 may include a networkinterface component 514 for sending and receiving data over a network,such as computer network or communications network. The apparatus 500may further include user input/output components 518, for example, atouchscreen display, or non-touch sensitive display with a keyboard,microphone, pointing device, or other separate user input device. Theapparatus 500 may optionally include a component for storinginformation, such as, for example, a memory device/component 516. Thecomputer readable medium or the memory component 516 may be operativelycoupled to the other components of the apparatus 500 via the bus 512 orthe like. The memory component 516 may be adapted to store computerreadable instructions and data for effecting the processes and behaviorof the components 502-504, and subcomponents thereof, or the processor510, or any of the operations of the methods disclosed herein. Thememory component 516 may retain instructions for executing functionsassociated with the components 502-504. While shown as being external tothe memory 516, it is to be understood that the components 502-504 canexist within the memory 516.

In view of the exemplary systems described supra, methodologies that maybe implemented in accordance with the disclosed subject matter have beendescribed with reference to one or more flow diagrams. While forpurposes of simplicity of explanation, the methodologies are shown anddescribed as a series of blocks, it is to be understood and appreciatedthat the claimed subject matter is not limited by the order of theblocks, as some blocks may occur in different orders and/or concurrentlywith other blocks from what is depicted and described herein. Moreover,not all illustrated blocks may be required to implement themethodologies described herein, or methodologies may include additionaloperations that are described herein without being illustrated.

The steps of a method or algorithm described in connection with thedisclosure herein may be embodied directly in hardware, in a softwaremodule executed by a processor, or in a combination of the two. Asoftware module may reside in RAM memory, flash memory, ROM memory,EPROM memory, EEPROM memory, registers, hard disk, a removable disk, aCD-ROM, or any other form of storage medium known in the art. Anexemplary storage medium is coupled to the processor such that theprocessor can read information from, and write information to, thestorage medium. In the alternative, the storage medium may be integralto the processor.

In one or more exemplary designs, the functions described may beimplemented in hardware, software, firmware, or any combination thereof.If implemented in software, the functions may be stored on ortransmitted over as one or more instructions or code on acomputer-readable medium. Non-transitory computer-readable mediaincludes computer storage media and temporary memory media. A storagemedium or memory medium may be any available medium that can be accessedby a general purpose or special purpose computer via an electronic,optical, magnetic, or other media reader. By way of example, and notlimitation, such computer-readable media may comprise RAM, ROM, EEPROM,CD-ROM, BluRay or other optical disk storage, magnetic disk storage orother magnetic storage devices, or any other medium that can be used tohold or store desired program code means in the form of instructions ordata structures and that can be accessed by a general-purpose orspecial-purpose computer, or a general-purpose or special-purposeprocessor.

The following numbered examples illuminate certain aspects of thepresent technology. The scope of the present technology may extendbeyond these examples in both details and generalities, as disclosed bythe foregoing description.

In addition to recognition of faces, locations, and text in still andmoving images, the change between video frame images may be utilized toidentify a person. Furthermore, motion may also be analyzed by radar,sonar, Forward Looking Infrared imaging, infrared imaging, audioanalysis (for example, of the sound of footfalls), and other analysistools. In addition, video is often accompanied by voice, which makespossible voice recognition.

Furthermore, such analysis may be utilized not only to identify aperson, but also to identify behaviors or even the truthfulness of aperson. As described at http://www.clarkfreshman.com, micro-expressionanalysis may be utilized to determine the likelihood that a person isbeing truthful. Automated facial micro-expression analysis is also beingdeveloped, as described in Yee-Hui et. Al. “A Survey of Automatic FacialMicro-Expression Analysis: Databases, Methods, and Challenges”,published in Frontiers in Psychology Vol. 9, 2018.

These inventions may be implemented in software that modifies video.Another method for implementation of certain of these inventions isclothing using a Clothing Display, which we define as a method to createthe illusion of movement, or to change, eliminate, or camouflagemovement, stance, or body appearance on a person (although calledClothing Display, it may also be used to create an illusion of movementor lack thereof on an object, and such use is included herein as well).The Clothing Display may comprise a flexible display, images projectedonto clothing or flesh, radar blocking or alteration technology, sonarblocking or alteration technology, projection of a person or objectbetween an observing camera and a person, or other methods of creatingan image on or near an object. It may also comprise clothing designed toabsorb and/or scatter sound, light or radiation. Although we call it a“Clothing Display” and use it herein in the context of a moving personor object and/or video of a moving person or object, certain aspects mayalso be used to block facial recognition. It should also be understoodthat “Clothing Display” may also comprise alteration of movement orother characteristics in video.

As described by Boyd and Little in a Biometric Gait Recognition in a2005 paper:

As a biometric, gait has several attractive properties. Acquisition ofimages portraying an individual's gait can be done easily in publicareas, with simple instrumentation, and does not require the cooperationor even awareness of the individual under observation. In fact, it seemsthat it is the possibility that a subject may not be aware of thesurveillance and identification that raises public concerns about gaitbiometrics.

Gait is, of course, the movements that result in locomotion. Human gaitis the movements that result in human locomotion. While this documentaddresses human gait, it should be understood to apply to animal gait aswell.

Human gait is thought to have at least three properties that are key tohuman perception and identification (as described by Boyd and Little):

-   -   1. Frequency: The components of the gait typically share a        common frequency;    -   2. Phase locking: The phase relationships among the components        of the components of the gait remain approximately constant. The        lock varies for different types of locomotion such as walking or        running;    -   3. Physical plausibility: The motion must be physically        plausible human motion.

In one aspect, a video of a person is obtained. For example, a systemmay be configured to impair automated gait analysis. Such a system wouldusually comprise a processor coupled to a memory, the memory encodedwith instructions that, when executed by the processor, cause the systemto analyze a video (and/or or sequential series of images). Those imagescould include objects, faces, bodies, animals and/or parts orcombinations thereof.

The system would identify the objects, faces, bodies, animals and/orparts or combinations thereof, and in one aspect may further identifyparts of one or more of the things identified. For example, with a videoof a human walking, the system may identify the joints and/or skeletonand/or arms, legs, torso and other elements.

Continuing with the example, the system may have identified the jointsand bones attached to those joints. Such identification may beaccomplished using the grassfire algorithm or other methods. Onceidentified, the image may be altered to modify the movement of joints,limbs, bones and/or other elements utilized or useful in identifying aperson, such as gait. The alteration may be made using the formula:

Obtain a minimal value of

$\Delta = {\sum\limits_{1toM}\overset{\_}{\partial i}}$

wherein ∂l is a measure of metric alteration each of for a number M ofdifferent movement (for example, one or more of the components thatcomprise gait). The method may further include selecting the number M ofmetrics so as to lower each applied ∂l to below a human-perceptiblethreshold. For example, the number M may be increased, thereby causingan increased number of alterations on a image, while the amount of eachalteration is decreased. To use a simple example, instead of changing anamount of arm swing by at least 10% to prevent automatic gaitrecognition (i.e., M=1 and ∂l>±10%), the alteration may change an amountof difference between right as opposed to left sided steps by a lesseramount, and may also change some other metric, for example the distancebetween the knees at various portions of a gait (e.g., M=2 and ∂l<±10%).

Such a modification would impair automatic recognition of the personwhile preserving an overall aesthetic quality of the image, wherein ∂lis a measure of metric alteration of each of a number M of the selectedfacial recognition metrics.

In one aspect, the methods described in Plagemann, Ganapathi, Koller andThrun in the paper “Real-time Identification and Localization of BodyParts from Depth Images” describes various methods to identify bodyparts in images, and such methods may be utilized to identify candidatesfor alteration.

If may further be desirable to modify captured moving and/or stillimages and/or sequences of images to hide, modify, and/or obscureelements that reveal cultural, behavioral, or other elements that may beundesirable. Alternatively, or in addition, desirable elements may beadded by modifying the video, image, or images.

Taking as an example an orthodox Jewish person wearing a yarmulke in apromotional video. While the promotional video may be entirelyappropriate to use in most of North America, imagine further that it isused as part of a presentation to a sovereign wealth fund where thesovereign has shown bias against Jews. In such a case, it may bedesirable to automatically delete the yarmulke from the video or stills,or even replace it with different headwear.

As another example, in certain cultures, showing the bottom of a shoe isconsidered an insult. A roundtable discussion where a participant hascross his legs and is revealing the sole of his shoe may be altered sothat a short, artificial podium, chair, or other object is placed infront of the participant to mask the showing of the bottom of the shoe.

As another example, it is common to show signs of physiologicalconditions in a video. As an example, it made national news in theUnited States when one of President Trump's invited guests at the 2019State of the Union address, Joshua Trump, fell asleep during theaddress. Similarly, yawning, flatulence, burping or other physiologicalstates may be captured in a still or video manner. The video may bemodified by identifying behaviors and/or body parts and/or theassociated movements, modifying the video and/or still image obscure orchange the undesirable behavior. Using the State of the Union example,the sleeping child may be replaced with video of the child capturedwhile awake, or the child may be removed from the video. In one aspect,the system may complete these actions by comparing the behaviors and/orother aspects of recognized images and/or persons to a database, andwhere the database indicates that the action is undesirable, the actionmay be automatically altered as described herein.

A further use of these inventions involves micro-expressions. Humans arewell known to have certain difficult (or impossible) to controlexpressions that accompany a mental state. As an example, expert ClarkFreshman was able to identify lies during a Presidential debate simplyby watching the footage for micro-expressions. As described athttps://www.huffingtonpost.com/entry/gop-debate-body-language_b_953564(retrieved Mar. 4, 2019), Professor Freshman looked for contempt, whichhe says is “somewhat easily detected by the simple smirk or sneer.” Oneinteresting characteristic of the contempt sneer is that it is “like asmile, but only half the face seems to get involved.” Professor Freshmanidentifies, for example, the contempt sneer on the face of Rick Perrywhen he was asked to comment on a report that whites have far morewealth than African-Americans. Similarly, a contempt half-smile wasgiven by Ron Paul when he criticized Governor Perry for an executiveorder to offer vaccines against HPV. As the Professor further noted,even changes in the blinking rate are revealing, showing that the fasterblinking rate indicates extra thinking.

Because these are micro-expressions, which are measured in milliseconds,they are very difficult for untrained humans to identify. However, itwould be an easy matter, as the Yee-Hui Oh paper demonstrates, toimplement a system that identifies some of these micro-expressions. If,for example, a candidate is announcing a run for President via aYouTube® video, or a public figure releases a video denyingparticipation in an affair or other undesirable revelation, automatedanalysis would enable various means of displaying where the bodylanguage draws the spoken words into question. One might imagine theword “contempt” flashed on the screen when contempt is detected.

In one aspect, the inventions may be utilized to obscure suchmicro-expressions, either by digitally maintaining the face (or other)muscles or components in place while they move, by freezing the image,by digitally moving the hand and arm to appear to scratch the cheek,blocking the view of the area showing the micro-expression, orotherwise. Conversely, micro-expressions might be added, whether insubstitution for contrary micro-expressions or on their own. Forexample, signs of truthfulness might be substituted for signs ofcontempt or lying. It is also desirable to use the formula disclosed inconjunction with the parent application to maintain a face that isrecognizable and where the alteration is not obvious. When removing amicro-expression, the portions of the face that did move as part of themicro-expression may be kept substantially in the same place relative tothe rest of the face. In one aspect, a new micro-expression may be addedwithout regard to whether an existing micro-expression is present.

In another aspect, voice analysis may be utilized to identify a person.In one aspect, an audio source, whether or not accompanied by video, maybe modified to alter the voice characteristics and therefore obscure theidentity of the person. When done in conjunction with a video, it ispreferred if mouth movements are altered to properly match any changesto voice cadence or other changes that would impact the synchronizationbetween the sound and the movement of the mouth and other imageelements.

One infirmity of the various methods, processes, and devices describedthroughout the specification (and not just in the “Continuation In Part”section) is that such of the image alteration (excluding thosealterations done in real time and projected or otherwise mapped onto thebody or object or face, as further described below) cannot beaccomplished as described without access to the video or photographicdata. That is, while a video or still image captured on a user's cellphone may be processed, a video or still image captured by agovernment-operated camera is put into the possession of the governmententity prior to any modifications made pursuant to the inventionsherein.

It is therefore desirable to modify the images captured by third partycameras (although it could be done for cameras in control of the user)by modifying how the user is seen (which we discuss this in conjunctionwith gait recognition, it should be understood that it may be utilizedwith any of the elements discussed herein, such as micro-expressions orface recognition). Such modifications may be done in a limited lightfrequency range, such as those areas just at the edge or just past theedge of human visual perception. These would comprise light outside (orclose to outside) of the 390 to 700 nanometer range. To be effective,the light would need to be within the range captured by most commercialcameras, which are typically capable of imaging light outside of thehuman spectral range.

Less bright light within the human spectral range may also be utilized,in order to minimize human perception of the light used to fool imagerecognition software. Alternatively, light could simply be used withoutregard to perceptibility by humans and/or could be camouflaged asfashion, camouflaged within other projected patterns, or otherwise.

In one aspect, joints and bones may be moved virtually by using LightProjection. We define Light Projection herein as using a modalitycapable of creating an image that overlays the person or object ofinterest and/or an area proximate to them. This can be accomplishedutilizing clothing with flexible displays or other means of displayingimages or light on clothing, and/or by using one or more pico projectorsor other projectors to project an image onto the person or object,and/or by other methods.

In another aspect, Light Projection may target the ground, walls, orother areas proximate to the person whose gait is to be obscured. Forexample, by using Light Projection on the ground around a person, acamera incapable of seeing three dimensions would have great difficultydistinguishing images of many walking people from the actual walkingperson. Similarly, images of many walking people may be projected onto awall where the person seeking gait anonymity may be walking near. In oneaspect, these projected images match the gait of somebody of greaterinterest than the subject (for example, a wanted criminal). In anotheraspect, those projected images match the gait of the subject and may beprojected traveling in different directions, preventing automated alertsas to the direction the subject is moving.

In another aspect, Light Projection may be used to project a differentgait onto a third party. For example, Joe may have a projector thatprojects his gait onto Jane's legs.

In one aspect, movement may be addressed in a manner where automatedgait (and other) analysis is substantially impaired by utilizingreflective, mirror-like clothing. In one aspect, the mirrors are locatedin at least one of the areas used for gait recognition. In anotheraspect, clothing elements are added that are capable of scatteringand/or obscuring radar and/or sonar imaging.

In another aspect, clothing may be heated and/or cooled using a patternthat makes heat-based (such as FLIR) image recognition unreliable. Forexample, the heat pattern of a face may be rendered below the arms of aperson, causing the system to believe that the gait is associated with aperson far shorter than the person of interest. Similarly, behaviorsassociated with heat, such as smoking, may be obscured from view atleast in part by causing the clothing to have “hot spots” that aresimilar to those associated with a burning cigarette. In oneimplementation, the hot spots may move, preventing an automated imagerecognition system from looking only for moving hot spots to identifysmoking behaviors.

In another aspect, mechanical elements may be placed on the body (in apreferred implementation, under at least one layer of clothing). Thoseelements may move in a random way for each step, in a systematic waythat is randomly set, in a systematic way that imitates characteristicsof another, or otherwise. The elements may be controlled by amicroprocessor. In another aspect, elements may be manually set andcontrolled.

The mechanical elements are preferentially (though not necessary) placednear joints and extend above and/or below the joint. To break automatedgait analysis, the mechanical joint is placed at a spot not directlyover the human joint. They are also preferentially (though notnecessarily) made to have a bendable joint-like element and/or haveextensions above and/or below the joint-like element. This allows theappearance of a joint that is moving in a different place and adifferent manner than the actual joint. Movement of the actual joint maybe obscured entirely using the mechanical elements and extensions belowand/or above the joint. Alternatively, movement of both joints may bevisible, creating substantial confusion in AI and other systems designedto do image recognition.

In another aspect, it may be desirable to exempt certain people fromimage and/or body and/or movement recognition. For example, the UnitedStates has preferred traveler programs and airport security programsthat allow users to undergo substantially less inspection andquestioning based on a pre-screening process. Persons who undergo apre-screening process may be given a method, such as an RFID tag, a baror QR code, a phone or device-borne transmitter, or may be placed in adatabase of faces and/or movements that are not to be tracked. Underthose circumstances, persons willing to be prescreened may preservetheir privacy.

What is claimed is:
 1. A system to impair automated moving imageanalysis, comprising: a processor coupled to a memory, the memoryencoded with instructions that when executed by the processor, cause thesystem to: analyze a digitally-encoded moving image comprising one ormore persons walking; apply a Clothing Display to at least one person toalter an apparent gait of the person using an alteration algorithm thatidentifies an alteration of gait metrics based on a value of:$\Delta = {\sum\limits_{1toM}\overset{\_}{\partial i}}$ to substantiallyreduce risk of automatic recognition of the apparent gait whilepreserving an overall aesthetic quality of the apparent gait, wherein ∂lis a measure of metric alteration of each of a number M of the gaitmetrics; and store in the memory, an altered digitally-encoded image. 2.The system of claim 1, where the Clothing Display comprises a flexibledisplay worn by a person.
 3. The system of claim 1, where the ClothingDisplay comprises images projected onto clothing of the person by atleast one projector.
 4. The system of claim 1, where the ClothingDisplay comprises projection of a person or object between an observingcamera and the person.
 5. The system of claim 1, where the ClothingDisplay comprises using at least one radar-scattering clothing elementworn by the person wishing to avoid gait analysis.
 6. The system ofclaim 1, where the Clothing Display comprises using at least onesonar-scattering clothing element worn by the person wishing to avoidgait analysis.